For purposes of performance or redundancy, modern storage systems can be built out of discrete components, called nodes, that are interconnected with a network, such as a TCP/IP network. Each node is typically a fully functional computer with CPUs, storage, memory, etc. The organization of nodes in such a system may be a peer-to-peer network, meaning all nodes are equal (i.e., there is no central management authority and no node is privileged). As equal peers, the nodes communicate among themselves to resolve state. Organizing a storage system as a peer-to-peer network can provide a more resilient and scalable storage solution, as nodes can be incrementally added for performance and/or capacity, and if one node fails, the storage system still works.
What distinguishes a peer-to-peer network storage system from just a collection of computers is that nodes in the system communicate with each other regarding the underlying data storage, health of each node, etc. Specifically, nodes in the system may copy and exchange information for purposes of performance and data integrity. This information may be in the form of data objects, or files, where an object may be a portion of a file.
As objects are propagated through the system, data structures are required to: a) know where the objects are; and b) know when the objects are no longer needed. In prior art systems where objects have reference counts, i.e., the number of times an object is referenced by another object or other data structure, an object may be safely de-allocated or deleted (e.g., garbage collection) when its reference count drops to zero.
However, keeping track of reference counts for billions of objects, when millions of objects are being passed around per second, results in unacceptable network traffic and CPU costs if simplistic algorithms are used.
Another challenge is determining that the reference count is actually zero, and then finding all object instances so they can be deleted. There is an ongoing need for more efficient and reliable protocols for tracking objects to overcome these problems.